The present invention relates generally to distributed file systems, and more particularly to distributed file systems optimized for time-critical sequential access to large files.
Distributed file systems include network nodes, which are computer systems attached directly to a network. Each network node has a processor, random-access memory (RAM), and an interface to a communication network. Nodes that are able to act as "servers" are interfaced to mass storage devices such as disk drives. The mass storage devices are usually partitioned in allocation units and data is read from or written to the device in multiples of sectors up to one allocation unit. In an access to a file on a given disk, the network node where the disk is located is called the "server" and the node from which the request was issued is called the "client." In a read access, data flows from the server to the client; in a write access, data flows from the client to the server. A single node may act as both client and server, and may run concurrent tasks. While one task waits, for example, for required data, other tasks can execute.
Because disk access is orders of magnitude slower than RAM access, large chunks of data may be read from the disk to RAM memory under the assumption that portions of, or near to, the data presently accessed have a high likelihood of being accessed in the near future. This is termed "caching." Caching may also be used with data to be written to the disk, to avoid the first disk access if the same data is later overwritten. A unit of cache data is said to be "dirty" if it has been modified in RAM memory, and therefore the corresponding data on disk must eventually be modified. The present invention uses read-ahead and write-behind caching techniques for sequential rather than repetitive file access, which attempt to separate the disk or network access from the read or write steps of application programs in order to lessen the need for waiting by the application program. In read ahead, future file read access by an application is predicted and the data is read into the cache before being requested by the application. In write behind, data to be written is placed into a cache and, after the application program resumes execution, written to the disk.
As distributed computing in multimedia, video, and audio data processing becomes widespread, the low speed of multi-megabyte file access over local area networks (LANs) becomes a significant problem. Even when high-speed networks, such as fiber distributed data interface (FDDI) token rings with data rates of 100 megabit per second (Mbps) are used, the combined latency (file access delay) due to client- and server-node task scheduling, their network-adapter data-transfer rates, the server disk latency and the network latency, has heretofore diminished the practicality of LANs for such data processing. For example, the throughput of Appletalk networks may be as low as 0.3 megabytes per second, and the throughput of Sun NFS (network file server) may be as low as 1 megabyte per second. To supply, in real time, sixteen sound channels requires a throughput of almost 2 megabytes per second.
As a result of this difficulty, a number of professional digital audio workstation manufacturers support the use of storage devices that are removable from fixed drives (removable media), such as disks based on magneto-optic (MO) technology for file transfers between workstations. This technology has been deemed necessary even though MO disk drives cost more than hard disk drives of comparable storage and have slower access times. Therefore, local hard disks are still used as the "workspace" for holding the data that is being worked with.
If a network rather than removable media is used for file transfers, multiple operators may work on different parts of the same project at the same time and concurrent access of multiple files is possible from each workstation or host computer. Accordingly, a primary object of the present invention is to provide a cooperative client-side and server-side file caching method that maximizes the end-to-end throughput of sequential file read and write access.
Another object of the present invention is to provide dynamic memory organization, reservation and allocation methods for competing memory-intensive activities.
A further object of the present invention is to provide a distributed file system with dedicated nodes capable of being connected to a workstation's memory bus and providing file server and client-side caching functions independent of the workstation.
Additional objects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the claims.